Do editors have a silver bullet? an agent-based model of peer review

This paper presents an agent-based model of peer review that looks at the effect of different editorial policies of referee selection. We tested four author/referee matching scenarios as follows: random selection of referees, selection of referees with a similar status to submission authors, selection of higher-skilled and lower skilled referees. We tested these scenarios against three types of referee behaviour, ie., fair, unreliable and strategic and measured their implications for the quality and efficiency of the process. Results show that in case of randomness of referee judgment, any editorial policy is detrimental for peer review. If referees behave strategically, certain matching policies, such as selecting referees of good quality, might counteract possible bias

Metodologia docent per a l’ensenyament dels fonaments de la informàtica en els graus no tècnics

La docència d’assignatures d’informàtica en titulacions no tècniques normalment requereix una adaptació de continguts que permeta cobrir la gran diversitat de coneixements de partida dels estudiants. En aquest article presentem una combinació d’estratègies d’aprenentatge i de mètodes d’avaluació que contribueix a anivellar els coneixements inicials de l’alumnat així com al desenvolupament de competències transversals. En particular, s’apliquen mètodes d’aprenentatge actiu en la docència dels continguts teòrics, per a involucrar l’alumne en el procés; s’empra l’aprenentatge basat en projectes en els laboratoris, per a incrementar-ne la motivació intrínseca a través de l’adquisició de coneixements pràctics fàcilment aplicables en la seua vida diària: i s’apliquen tècniques d’aprenentatge cooperatiu, per a desenvolupar l’habilitat de treball en grup. Els resultats obtinguts mitjançant l’aplicació d’aquesta estratègia mostren un impacte positiu sobre els resultats acadèmics de l’alumnat.Delivering Computer Science modules in non technical degrees usually requires adapting contents to very diverse student backgrounds. In this article we present a combination of teaching strategies and evaluation methods which contribute to levelling the initial students’ knowledge and developing competences and transferrable skills. In particular, active learning methodologies are applied in the delivery of theoretical contents, to get the student involved in the learning process; project based learning is used in practical laboratories, to increase their intrinsic motivation by acquiring valuable hands-on experience which can easily be applied in their daily activities; problem based learning is used to promote autonomous learning; and cooperative learning techniques are used to develop team work abilities. Results show that the application of this technique has had a positive impact on the students’ academic results.Versión corregida en valenciano del artículo publicad

When competition is pushed too hard. An agent-based model of strategic behaviour of referees in peer review

This paper examines the impact of strategic behaviour of referees on the quality and efficiency of peer review. We modelled peer review as a process based on knowledge asymmetry and subject to evaluation bias. We built two simulation scenarios to investigate largescale implications of referee behaviour and judgment bias. The first one was inspired by "the luck of the reviewer draw" idea. In this case, we assumed that referees randomly fell into Type I and Type II errors, i.e., recommending submissions of low quality to be published or recommending against the publishing of submissions which should have been published. In the second scenario, we assumed that certain referees tried intentionally to outperform potential competitors by underrating the value of their submissions. We found that when publication selection increased, the presence of a minority of cheaters may dramatically undermine the quality and efficiency of peer review even compared with a scenario purely dominated by "the luck of the reviewer draw". We also found that peer review outcomes are significantly influenced by differences in the way scientists identify potential competitors in the system

A simulation of disagreement for control of rational cheating in peer review

Understanding the peer review process could help research and shed light on the mech-anisms that underlie crowdsourcing. In this paper, we present an agent-based model of peer review built on three entities - the paper, the scientist and the conference. The system is implemented on a BDI platform (Jason) that allows to dene a rich model of scoring, evaluating and selecting papers for conferences. Then, we propose a programme committee update mechanism based on disagreement control that is able to remove reviewers applying a strategy aimed to prevent papers better than their own to be ac-cepted (rational cheating"). We analyze a homogeneous scenario, where all conferences aim to the same level of quality, and a heterogeneous scenario, in which conferences request dierent qualities, showing how this aects the update mechanism proposed. We also present a rst step towards an empirical validation of our model that compares the amount of disagreements found in real conferences with that obtained in our simulations

Integració d'habilitats socials en l'animació comportamental d'actors sintètics.

RESUMEN La simulació de mons virtuals habitats per personatges 3D és un problema complex que necessita la integració d'àrees diverses com ara els gràfics per computador i la intelligència artificial. De la primera, hom pot obtenir la credibilitat gràfica de l'escena, de la segona, l'autonomia i la interactivitat pròpia de l'animació comportamental dels així coneguts com 3DIVA (3D Intelligent Virtual Agents). Normalment, la interactivitat dels 3DIVA s'adreça a l'entorn (p. ex. la capacitat d'executar tasques sobre els objectes 3D) o a l'usuari (p. ex. la capacitat de comunicació amb un usuari humà, la reproducció d'emocions o actituds expressives, etc.). Amb tot, cal fer una passa endavant quan es treballa amb societats artificials com les que apareixen, per exemple, dins dels mons virtuals dels jocs d'entreteniment, dels simuladors per a l'aprenentatge de tasques civils o militars, o de ciberespais com el famós SecondLife. Malgrat el realisme gràfic assolit per les tècniques d'animació, és necessari aprofundir en la interacció entre aquests actors sintètics autònoms per tal de fer-los evolucionar cap a una intelligència social que estiga més a prop de la sociabilitat humana. L'objectiu principal d'aquesta tesi és la integració d'habilitats socials en l'animació comportamental d'humanoides autònoms situats dins de mons virtuals 3D. Per tant, en primer lloc fem una revisió dels mons virtuals habitats, així com dels diferents mecanismes d'animació comportamental emprats normalment en actors sintètics. En segon lloc, repassem els models de sociabilitat més comuns que hom pot trobar dins l'estat de l'art dels agents socialment intelligents. La presa de decisions que han de dur a terme els actors sintètics amb capacitats socials per interaccionar amb la resta d'agents comporta processos cognitius complexos i requereix un coneixement abstracte dels elements de l'entorn. Darrerament, hom ha proposat la inclusió d'informació semàntica dins dels entorns virtuals, coneguts sota el terme anglosaxó de Semantic Virtual Environments (SVE). D'acord amb açò, en aquest treball proposem l'ús d'ontologies per aconseguir tres objectius: millorar la sensorització d'escenes complexes, definir operatives generals que els agents puguen reutilitzar en situacions diverses i definir les relacions socials establides entre els membres d'una societat artificial. La natura proactiva dels actors sintètics ha fet que les tècniques d'animació comportamental més comunes, basades en la selecció dinàmica d'accions, es divideixen fonamentalment en dos grups: la planificació de tasques (p. ex. les basades en STRIPS) i els sistemes basats en regles (p. ex. els models Belief-Desire-Intention). Les aportacions principals d'aquesta tesi es troben en la inclusió d'habilitats socials en ambdós paradigmes, ja que els considerem aproximacions vàlides extensament emprades i moltes vegades complementàries. Primerament, presentem com aconseguir comportaments collaboratius en grups d'agents basats en planificadors heurístics. Fem servir la comunicació d'accions per a la coordinació entre les activitats dels humanoides i la comunicació d'objectius per a la cooperació entre els personatges 3D. Llavors, hem desenvolupat un mecanisme de sospesament heurístic que permet assignar un pes a cada objectiu, de manera que reflectisca la seua importància social. No obstant això, l'eficiència no és l'única manera d'expressar sociabilitat. Pel contrari, la presa de decisions humana sovint té en compte més d'un punt de vista i no tots persegueixen la consecució ràpida de les tasques encarregades. Per tal causa, presentem MADeM (Multi-modal Agent Decision Making), un procés de presa de decisions de tipus social que pot ser emprat per agents BDI per tal de prendre decisions socialment acceptables. MADeM és capaç davaluar diferents solucions a un problema mitjançant tècniques socials basades en leconomia de mercats les quals fan servir les subhastes com a mecanisme perquè els agents expressen les seues preferències. __________________________________________________________________________________________________The aim of this thesis is to integrate social skills in the behevioural animation of humanoids inhabiting 3D virtual worlds. In order to behave socially, synthetic actors must be provided with complex decision making mechanisms able to manage abstract knowledge in the environment. Semantical Virtual Environments have recently been proposed not only to ease world modelling but also to enhance agent-object and agent-agent interaction. Thus, we propose the use of ontologies to: i) introduce semantic levels of detail that help the sensorization of complex scenes; ii) create general and reusable operativity for autonomous characters; and iii) define social relations among agents within an artificial society. Essentially, two approaches have been followed when dynamically selecting the actions being performed by proactive virtual actors: task planning (e.g. STRIPS) and rule based systems (e.g. BDI). According to this, we have developed two techniques to incorporate social skills in both paradigms. First, we show how to obtain collaborative behaviours in groups of humanoids based on heuristic search planners. We use action communication to coordinate the agent's activities and goal communication to generate cooperation among the 3D characters. Then, we have developed an heuristic weighting allowing to associate a weight to each goal in order to reflect its social importance. However, efficiency is not the only way of expressing sociability. Instead, human decision making usually considers different points of view. Therefore, we present MADeM (Multi-modal Agent Decision Making), a social agent decision-making proces that BDI agents can use to perform socially acceptable decisions. MADeM uses auctions as the mechanism to get agent preferences (expressed by utility functions) and it is able to evaluate different solutions to a certain problem by means of a social technique based on welfare economics

Towards reactive navigation and attention skills for 3D intelligent characters

This paper presents a neural design which is able to provide the necessary reactive navigation and attention skills for 3D embodied agents (virtual humanoids or characters). Based on Grossberg’s neural model of conditioning [6], as recently implemented by Chang and Gaudiando [7], and according to the Adaptative Resonance Theory (ART) and the neuroscientific concepts associated, the neural design introduced has been divided in two main phases. Firstly, an environmentcategorization phase, where an on-line pattern recognition and categorization of the current agent sensory input data is carried out by a self organizing neural network, which will finally provide the agent’s short term memory layer(STM). Secondly, and based on the classical conditioning paradigm, the model will associate the interesting STM states, from the navigation or attention points of view, to finally simulate these necessary skills for 3D characters or humanoids. Finally, we will show some experimental navigational results, through the integration of the model presented in 3D virtual environments.Partially supported by the GVA-project CTIDIB-2002-182 (Spain)

New frontiers of peer review

This news article introduces a new COST Action entitled PEERE (TD1306), which stands for New Frontiers of Peer Review (PEERE). PEERE is a trans-domain proposal which brings together researchers from various different disciplines and science stake-holders for the purpose of reviewing the process of peer review. PEERE officially began in May 2014 and will end in May 2018. Thirty-one countries, including Malta, are currently participating in the Action. In order to set the context in which this COST Action was initiated, we first look very briefly at the history of the process of peer review and various models of peer review currently in use. We then share what this COST Action hopes to achieve.peer-reviewe

Assessing peer review by gauging the fate of rejected manuscripts: the case of the Journal of Artificial Societies and Social Simulation

This paper investigates the fate of manuscripts that were rejected from JASSS-The Journal of Artificial Societies and Social Simulation, the flagship journal of social simulation. We tracked 456 manuscripts that were rejected from 1997 to 2011 and traced their subsequent publication as journal articles, conference papers or working papers. We compared the impact factor of the publishing journal and the citations of those manuscripts that were eventually published against the yearly impact factor of JASSS and the number of citations achieved by the JASSS mean and top cited articles. Only 10% of the rejected manuscripts were eventually published in a journal that was indexed in the Web of Science, although most of the rejected manuscripts were published elsewhere. Being exposed to more than one round of reviews before rejection, having received a more detailed reviewer report and being subjected to higher inter-reviewer disagreement were all associated with the number of citations received when the manuscript was eventually published. This indicates that peer review could contribute to increasing the quality even of rejected manuscript

A few bad apples are enough: an agent-based peer review game

Following previous agent-based research on peer review, this paper presents a game theory-inspired model that looks at peer review as a cooperation dilemma. We tested different scientist behaviours and network topologies in order to understand their implications on the quality, efficiency and type of resource distribution in the science system. We tested random, scale-free and small world networks connecting scientists and three types of referee behaviour: self-interested (providing unreliable opinion), normative referees (providing reliable opinion) and conformist reviewers (conforming to other referees' behaviour). Preliminary results indicate that differences in the combination of referee behaviour have significant impact on the quality of the process and that the percentage of conformists is one of the most crucial model parameters

The Invisible Hand of Peer Review: The Implications of Author-Referee Networks on Peer Review in a Scholarly Journal

Peer review is not only a quality screening mechanism for scholarly journals. It also connects authors and referees either directly or indirectly. This means that their positions in the network structure of the community could influence the process, while peer review could in turn influence subsequent networking and collaboration. This paper aims to map these complex network implications by looking at 2232 author/referee couples in an interdisciplinary journal that uses double blind peer review. By reconstructing temporal co-authorship networks, we found that referees tended to recommend more positively submissions by authors who were within three steps in their collaboration network. We also found that co-authorship network positions changed after peer review, with the distances between network neighbours decreasing more rapidly than could have been expected had the changes been random. This suggests that peer review could not only reflect but also create and accelerate scientific collaboration